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ABSTRACT 

The findings from recent Canadian research indicate that while agency managers and 
front-line workers agree that evaluation is important, they seldom evaluate their work 
with clients. With the current emphasis on evidence-based practice and outcome-focused 
intervention, it is important to be able to demonstrate the value of career services in a 
manner that service providers find meaningful and funders find useful. In this article a 
framework for evaluation is presented that permits linking the services provided with the 
client outcomes that are being achieved. 

RESUMfi 

Les resultats de recherches canadiennes recentes indiquent que les directeurs d’organismes 
et les travailleurs de premiere ligne evaluent rarement leur travail aupres des clients meme 
s’ils conviennent de fimportance d’une evaluation. Vu l’accent mis actuellement sur la 
pratique et l’intervention axees sur les resultats, il est important de pouvoir demontrer 
la valeur des services de perfectionnement professionnel d’une fa^on que les fournisseurs 
de services trouvent signifiante et que les bailleurs de fonds trouvent utile. Cet article 
presente un cadre devaluation qui permet de relier les services fournis aux resultats realises 
aupres des clients. 
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Recent Canadian research (Lalande, Hiebert, Magnusson, Bezanson, & Borgen, 
2006; Lalande & Magnusson, 2007) indicates that front-line workers and agency 
managers believe that it is important to evaluate career guidance services; however, 
the majority reported that they seldom actually evaluate their work with clients. 
Furthermore, the majority of respondents indicated that they thought the work 
they did was beneficial and effective, but the evidence they collected would make 
it difficult to provide a convincing link between the interventions they used and 
the outcomes they observed. 

In international symposiums held in 1999, 2001, and 2003 (Bezanson & 
O’Reilly, 2002; Hiebert & Bezanson, 2000), as well as in a national symposium 
held in 2004, a common theme expressed by policy makers was that they were 
amenable to consider providing funds for career development services, but, in 
the absence of evidence attesting to the efficacy of such services, it was difficult 
for them to justify spending the money. There was a frequently occurring theme 
of show me the evidence that for the most part came from policy makers, and was 
met with resistance from service providers. However, the challenge is legitimate: 
Without efficacy evidence, it is difficult for funders to support the delivery of 
career development services. 

After the 2005 international symposium, a group of researchers decided to 
explore the manner in which the challenge of show me the evidence could be 
addressed. An exploratory meeting was held, funded by Human Resources and 
Social Development Canada (HRSDC) and facilitated by the Canadian Career 
Development Foundation (CCDF). From that initial meeting a group of research- 
ers formed the Canadian Research Working Group for Evidence-Based Practice in 
Career Development (CRWG) to address the need to demonstrate the efficacy of 
career development services. The evaluation framework presented in this article 
is the result of collaboration between members of the CRWG. 

A DRAFT EVALUATION FRAMEWORK 

There are many good evaluation models in the literature (cf. Fitzpatrick, 
Sanders, & Worthen, 2004). However, no one evaluation model is “best” in all 
regards: Each model has strong points and each has aspects that are not as suit- 
able for the career development field. In early meetings, members of the CRWG 
examined numerous evaluation models and concluded that since no single model 
was superior in all regards, it was best to pick one model and develop it as fully as 
possible. In developing an evaluation framework, members of the CRWG were 
cognizant of three guiding principles: (a) A framework is simply a way of organ- 
izing relevant information, there likely is no best framework, and any number of 
frameworks would likely be sufficient for most tasks; (b) The underlying purpose 
of any framework is to organize relevant information in a way that permits people 
to look for causal connections, or areas of influence, between the types of informa- 
tion collected; and (c) It is useful to select a framework that is easy to understand 
(not overly complicated) and yet robust enough to accommodate most of the data 
that are relevant to career development services. 
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As a starting point, members of the CRWG chose a very basic framework 
consisting of three elements: inputs, processes, and outcomes. The framework is 
focused on understanding the changes (outcomes) experienced by people receiv- 
ing career services. Inputs refer to the resources that are available to help clients 
change (i.e., pursue the outcomes). Processes are the mechanisms that are involved 
in achieving the outcomes. Outcomes are the changes in service recipients (clients), 
that is, the results of the inputs enacting the processes. These three components 
are elaborated below and depicted pictorially in Figure 1. 


Figure 1 

Evaluation framework: A tool for organizing relevant evaluation information 


Processes 


Outcomes 


Resources available 

• Staff 

— Number of staff, 
level of training, 
type of training 

• Funding 

- Budget (funding 
level) 

• Service guidelines 

- Agency mandate 

• Facilities 

• Infrastructure 

• Community resources 


Activities and process 
that link to outputs or 
deliverables 

1 . Generic interventions 

• Working alliance, 
microskills, etc. 

2. Specific interventions 

• Interventions used by 
service providers 

- Skills or strategies 
used by service 
providers 

- Home practice 
completed by clients 

• Programs offered by 
agency 

• Involvement by third 
parties 

• Quality of service 
indicators 

- Stakeholder 
satisfaction 


Indicators of client change 

1 . Learning outcomes 

• Knowledge and skills 
that can be linked 
directly to the program 
or intervention being 
used 

2. Personal attribute out- 
comes 

• Changes in attitudes 

• Intrapersonal variables 
such as self-esteem, 
motivation, etc.) 

3. Impact outcomes 

• Impact that the learn- 
ing outcomes or the 
personal attribute 
outcomes have on the 
client’s life, such as 
employment status, 
enrolled in training, 
etc. 

• Social and relational 
impact 

• Economic impact 
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Outcomes 

In the model depicted in Figure 1, outcomes refer to the specific results of 
an intervention, including changes in client competence (knowledge and skills), 
changes in client personal attributes, changes in client situation, and/or broader 
changes for the client and/or community. These broader outcomes usually can be 
thought of as the impact of the changes in client competence or changes in client 
personal attributes. The choice to tie outcome to client change was deliberate. 
Research conducted in Alberta (Hiebert, Lalande, & Magnusson, 2005) and na- 
tionally (Lalande et al., 2006; Lalande & Magnusson, 2007) indicated that service 
providers had difficulty identifying outcomes that could be linked to programs or 
services offered. For example, research participants suggested that an intervention 
such as networking could result in outcomes such as build and maintain a posi- 
tive personal image, change and grow throughout one’s life, and maintain balanced 
life/work roles. Clearly, the link between teaching clients to network and building 
a positive self-image is tenuous at best. Furthermore, few agencies, we suspect, 
would feel comfortable being held accountable for producing outcomes such as 
building a positive self-image as a result of a workshop on networking. Thus, we 
have restricted our conceptualization of outcome to client change in knowledge 
or skills, personal attributes, and life impact. 

Framework for organizing client learning outcomes. As a means of organizing the 
things that clients might learn as a result of accessing career services, the CRWG 
suggests using the framework from Blueprint for Life/Work Designs (Hache, Rede- 
kopp, & Jarvis, 2000). The Blueprint contains 1 1 categories of knowledge and/or 
skills grouped into three clusters (see Figure 2). The Blueprint's become widely 
accepted as a way of describing client learning outcomes and many of the resources 
available to career practitioners are now indexed to the Blueprint, making it easier 
to choose resources that are appropriate for the learning outcomes being sought. 

Figure 2 

The Blueprint for Life/Work Designs: A framework for organizing client learning 
outcomes 


1 . Personal management outcomes 

• Build and maintain a positive personal image 

• Interact positively and effectively with others 

• Change and grow throughout one’s life 

2. Learning and work exploration outcomes 

• Participate in lifelong learning supportive of life/work goals 

• Locate and effectively use life/work information 

• Understand the relationship between work and society/economy 

3. Life/work building outcomes 

• Secure, create, and maintain work 

• Make life/work-enhancing decisions 

• Maintain balanced life and work roles 

• Understand the changing nature of life/work roles 

• Understand, engage in, and manage one’s own life/work process 
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Framework for organizing client personal attribute outcomes. In the research cited 
above (Hiebert et al., 2005; Lalande et al., 2006; Lalande & Magnusson, 2007), 
respondents identified some changes in client personal attributes that they thought 
were important, but which they did not (or were not allowed to) measure or re- 
port. Rather than get into a debate about whether or not these personal attributes 
are in fact learned, we have identified them as a separate category of outcomes, 
acknowledging that some people may view these as learning outcomes and others 
may not. We have organized these personal attributes into three broad categories: 
(a) attitudes (e.g., belief that change is possible, internal locus of control); (b) 
intrapersonal factors (e.g., confidence, motivation, self-esteem); and (c) client 
independence (e.g., client self-reliance, client initiative, independent client use 
of tools provided in career services). 

In the literature these personal attributes are sometimes referred to precursors 
(Hiebert, 1994; Killeen, White, & Watts, 1993; Maguire & Killeen, 2003), for 
they often mediate between skill and knowledge attainment and the life-impact 
outcomes. These personal attributes speak to a client’s willingness (ability) to 
put learning into action. In one of the first meetings of the CRWG, agreement 
was reached that these are in fact outcomes, and the suggestion was made that 
we cluster them together in a category called “personal attributes.” Regardless of 
the label used to describe these sorts of outcomes, there is widespread agreement 
that they are important and that it most often is difficult to obtain the desired 
amount of impact if these personal attributes are not addressed. Therefore, these 
types of client outcomes need to be identified and addressed in their own right, 
and trustworthy methods need to be developed to evaluate these variables. 

Framework for organizing client impact outcomes. Impact outcomes are the spin- 
off effects that derive from the learning outcomes, and perhaps from the personal 
attribute outcomes. They are the ultimate, hoped-for end result of an intervention. 
They include things such as employment status, placement rates, participation in 
training, engaging in job search, client ability to fit in at the workplace (resulting 
in increased job stability), societal impacts (such as reduced crime or reduced 
substance abuse), relational impacts (e.g., better relations with co-workers, or 
spouse), and economic impacts. 

Agencies and funders need to collaborate and reach agreement on what im- 
pacts can be reasonably expected. For example, in times of high unemployment it 
might not be reasonable to expect that all people who participate in a work search 
program will end up finding employment. In times of lower unemployment, or 
in geographic regions where the job market is robust, it might be more reasonable 
to expect high placement rates following completion of a work search program 
and also high retention rates for 6 months or 12 months because clients found 
personally meaningful work rather than accepting the first job offer. 

Processes 

Processes can be thought of as having two main components: interventions, 
which can be linked directly to the outcomes being sought, and quality service 
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factors, which have an effect on the general operation of the agency but do not 
directly relate to any specific outcomes. In the framework depicted in Figure 1, 
intervention refers to any intentional activity implemented in the hopes of foster- 
ing client change. Interventions include the interactions of service providers with 
clients or third parties as well as the components of the programs and services that 
are instrumental in achieving the client outcomes. 

We conceptualize two broad categories of interventions: generic and specific. 
Generic interventions are those actions that are common to most interactions 
between service providers and clients or third parties. For example, a strong work- 
ing alliance between service provider and client has been shown to be important 
in facilitating client change, and developing a working alliance likely will be part 
of virtually all interventions. Similarly, teaching a client to reframe an unpleasant 
event and view it as an opportunity for growth could be part of several interven- 
tions and could contribute to achieving numerous client outcomes. Specific in- 
terventions are more singularly focused than generic interventions. They usually 
are linked directly to client goals and outcomes, or linked to interactions with 
third parties that are intended to foster client change. Specific interventions can 
be part of interactions with clients directly or they can be bundled together as 
part of programs and workshops. 

A framework for organizing specific interventions. Based on standard practices 
and reports from the field, we suggest a framework for organizing the specific 
interventions that are used to initiate and/or sustain client change (see Figure 3). 
The first four categories (career decision making, skills enhancement, job search, 
and job maintenance) represent typical interventions used to achieve learning 
outcomes, although some of the bullet points depict client personal attributes 
that usually are by-products of learning the knowledge or skills involved. The fifth 
category, career-related personal development, pertains to interventions that are 
designed explicitly to address changes in personal attributes. We have included 
the term “career-related” in the title to acknowledge that most agencies offering 
career development services do not have a mandate to offer personal counselling, 
but when there is a personal issue, such as lack of self-confidence, that is affect- 
ing a client’s ability to pursue career goals, it is important to address that issue 
as part of the career intervention. The sixth category is for interventions that do 
not neatly fit into the other five categories: it is our way of acknowledging that 
an important and legitimate part of providing comprehensive career develop- 
ment services is to be able to recognize when someone may need to be referred, 
for example, to a detox program prior to beginning work aimed more directly at 
finding employment. 

The categories in Figure 3 represent topics that are addressed in specific inter- 
ventions. We have used topical headings as our organizing tool, recognizing that 
the topics might be addressed in programs or other services in a variety of different 
ways, for example, through counselling, teaching, workshop facilitation, guidance, 
and made available in a group setting, a classroom, through individual counsel- 
ling, or guided self-help. Furthermore, different agencies may decide to address 
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some topic with different delivery mechanisms. The mandate of the agency, the 
expertise of the staff, and the learning styles of the clients typically all come into 
play when deciding how the interventions are implemented. 

Figure 3 

Framework for organizing client change interventions 

1. Career decision making 

• Self-awareness of occupational interests, relevant aptitudes, personal values, job-related skills, 
adaptability skills 

• How to find and use labour market information 

• Knowledge of community resources 

• Awareness of alternative employment options 

• Adaptability in range of options considered 

2. Work-specific skills enhancement 

• Specific employment skill training (certificate or training program completed) 

• Skills for creating opportunities 

• Self-management skills 

• Literacy skills 

• Numeracy skills 

• Computer skills 

3. Work search 

• How to identify and follow-up on job leads 

• Preparing resume and cover letter 

• Self-presentation skills 

• Job interview skills 

• Ability to negotiate appropriate employment contract 

4. Job maintenance 

• Acquiring information about job entitlements 

• Knowledge of factors contributing to job loss 

• Conflict resolution skills 

• Overall work habits 

5. Career- related personal development 

• Building self-efficacy (e.g., self-confidence, motivation, belief that change is possible) 

• Skills for managing life demands (e.g., stress control, time management, financial management, 
interpersonal skills, coping with psychological effects of job loss) 

• Managing transitions (e.g., flexibility and adaptability) 

• Becoming more aware of employment opportunities 

• Becoming more self-directed 

6. Other 

• Referral 

• Psychological assessment 


Quality service factors. Even though quality service factors do not link directly 
to client outcomes, they are important to address. To illustrate, a client who gets 
rude treatment from a receptionist and encounters a dismissive attitude by a 
group facilitator will likely not achieve the best outcomes, even from a very good 
program. Thus, even though some processes cannot be linked directly to client 
outcomes, they are important to track. The list of such factors that agencies and 
practitioners expressed a desire to measure include the following: client satisfaction, 
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stakeholder satisfaction, employer satisfaction, level of service utilization, number 
of applicants for services, client relationship with the agency (clients return for 
service, clients are self reliant, etc.), time required for service provision, agency 
reputation, and ability to fund-raise. Quality service factors are sometimes thought 
of as outcomes; however, they are not indicators of client change, per se. Therefore, 
we suggest that they be regarded as indicators of successful processes. 

One difficulty in using an “Inputs — > Processes — > Outcomes” model is de- 
ciding where to put outputs. Outputs usually refer to things such as quality of 
service, client flow, scope of services, types of client problems, and so on. If we 
include these sorts of variables in the “outcomes” category, they get confused with 
indicators of client change. Therefore, we suggest including them in “processes” 
as they are not really indicators of client change, but indicators that appropri- 
ate processes have been followed. Thus, processes become all the activities that 
agencies, clients, third parties, and so on engage in; to evaluate the processes, we 
gather evidence on counsellor and client adherence to the intervention plan, and 
also client satisfaction with the service, agency adherence to mandate, numbers 
of clients seen, and so on. 

Inputs 

Inputs refer to the resource base that an agency can access to deliver career 
services. The amount and scope of resources that an agency can access has a large 
influence on the agency’s ability to offer quality services. A framework for organ- 
izing input variables is not elaborated in this article, because those factors were 
not assessed in the national survey or the Alberta study cited earlier. The system 
for organizing inputs undoubtedly will be revised and elaborated as the draft 
evaluation framework is field tested. However, we offer as a starting point the 
following framework for tracking inputs: staff (number of staff, competencies of 
staff, staff level of training, type of training), funding level, infrastructure (facili- 
ties, support staff, consultants), and community resources (other professionals, 
physical resources such as libraries, internet cafes, etc.). Documenting inputs is 
an important part of demonstrating cost-effective service delivery and making 
sure that an agency had a resource base adequate to achieve the outcomes that 
are expected. 


USING THE DRAFT EVALUATION FRAMEWORK 

The three elements in Figure 1 can be thought of as related in a linear way, but 
the relationship is not strictly linear. Inputs feed processes and processes result in 
outcomes. However, outcomes are also influenced by the inputs available, and the 
nature of the inputs (especially the competencies of the staff) influence the process 
that can be enacted. Thus, even though Figure 1 depicts a linear relationship, in 
reality the three elements are very interactive. 

In order to perform a comprehensive evaluation, all three elements need to be 
examined and evidence needs to be gathered on all three evaluation components. 
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The resources that are available (inputs) have an effect on what outcomes are 
realistic — both the nature of the outcomes and volume of outcomes that can be 
obtained. Thus, an evaluation plan needs to describe the inputs (both the nature 
of the inputs and the amount available) and link the inputs to the processes fol- 
lowed. Evidence needs to be gathered on the processes that were enacted so that a 
link can be made between the outcomes obtained and the processes that were used 
to obtain those outcomes. Ultimately, it is useful to be able to have evidence that 
will support a claim that these outcomes were obtained because those processes 
were followed, and we could follow those processes because we had such and such 
in our resource base. Agencies that routinely gather these sorts of data have an 
easy time addressing accountability concerns, especially if the evidence gathering 
is integrated into service delivery. 

An important part of evidence-based practice involves documenting the proc- 
esses followed by both service providers and service recipients, as well as any sig- 
nificant others that potentially could be affecting the achievement of outcomes. In 
order to claim that an intervention is responsible for producing a client outcome, 
it is important to be able to say with confidence that both service provider and 
client have followed the intervention plan. There are many examples in the ca- 
reer development field where an intervention appears to be ineffective, but closer 
scrutiny reveals that the intervention plan was in fact not followed. 

An important next step in developing a comprehensive evaluation plan is to 
map the client learning outcomes onto the interventions. Agencies will need to 
be very clear about the client outcomes that they want to be held accountable 
for, and then to identify where in the programs or services they offer the compo- 
nents are that are likely to produce those outcomes. This mapping will need to 
be quite detailed and likely will be quite specific to an agency or a program, but 
using a common framework to identify the outcomes and the interventions will 
help to identify what outcomes are reasonable to expect from the services being 
offered. 

Finally, when developing programs, the logical place to begin is by identifying 
the outcomes that are desired, then planning the processes that will be needed to 
produce the outcomes. After the processes are elaborated, it is sensible to look at 
the inputs that will be needed in order to implement the process. Some agencies 
may have limitations on their inputs, especially in matters relating to budget or 
to the credentials of their service providers, including the other community re- 
sources that are available. Those sorts of factors may restrict the interventions or 
programs that can be implemented, and subsequently the outcomes that can be 
achieved; by implication, it may also mean that additional resources will be needed 
in order to implement a program successfully. We emphasize this point because 
programs are often designed without explicit expectations for the outcomes that 
are intended to be achieved. We believe that the program development process 
should begin by articulating the client needs that are intended to be met by the 
program, then identifying the specific types of client changes that are intended to 
result from participation in the program (i.e., the intended client outcomes). After 
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the outcomes are clear, then the program content and accompanying processes 
can be developed and the evaluation plan can be elaborated. 

Implementing an evaluation model like the one described in this article will 
involve a shift in thinking for many agency managers and service providers. To 
demonstrate the value of career services and address the “show me the evidence” 
challenge, the question “What are we going to do?” will always need to be asked 
in conjunction with the question “How will we tell how well it is working?” 
Thus, the program plan and the evaluation plan need to be developed together 
and implemented at the same time. This is somewhat of a departure from normal 
practice in many circles, but is likely a necessary consideration when the focus is 
on evidence-based practice and outcome-focused intervention. 

SUMMARY 

In the research described in the opening pages of this article, respondents 
expressed frustration that client contextual factors and societal impacts typically 
were not seen as relevant for evaluation. Respondents also identified many impor- 
tant outcomes that went unmeasured or unreported. Funders and policy makers 
acknowledged that career services were important, but lamented that there was 
insufficient evidence to support the funding levels that were being requested. 
The evaluation framework described in this article has the potential to satisfy the 
concerns of both parties, because it encompasses a focus on both evidence-based 
practice and outcome-focused intervention. We believe that the model is easy to 
understand, robust enough to encompass most aspects of career development 
services, and easily incorporated into the practices of agencies and service provid- 
ers. Field testing is currently in progress with a number of agencies, and reports 
on the adequacy of the framework will begin to appear in the near future. 

In closing, we reiterate a point made earlier, that any comprehensive plan for 
evaluating the effectiveness of career development interventions needs to incor- 
porate a systematic method for gathering data on all three components of the 
framework we are proposing. In order to get an accurate picture of which inter- 
ventions work best, with which clients, under which circumstances, it is necessary 
to examine the client outcomes, the processes used to obtain those outcomes, and 
the inputs needed to enact the processes. 

Please note that the framework presented in this article is a draft, intended for 
discussion and further refinement by the career development community. In field 
testing the framework described in this article , we particularly want users to ask 
themselves “How will this approach fit in our work place?” and “What changes 
need to be made in the framework in order for it to more adequately address 
the unique needs of our work place?” We expect that the organizing categories 
in the framework will change as the framework is tested in the field. Thus, this 
is a dynamic framework that is very much a work in progress, and one that the 
CRWG will revise as needed to accommodate user feedback and the realities that 
exist in the field. 
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